[FLINK-34930] Fork existing code from bahir-flink #1
Conversation
ferenc-csaky
force-pushed
the
FLINK-34930
branch
2 times, most recently
from
d62ac38
to
82f84c8
Compare
|
@MartijnVisser can you trigger another CI run pls? |
ferenc-csaky
force-pushed
the
FLINK-34930
branch
from
82f84c8
to
3c0edbe
Compare
When running docker based integration tests locally, fail silentily if env requirements not available. Closes #38 Closes #35
Closes #48
By default, KuduSink processing message one by one without checkpoint. When checkoint is enabled, throughput is improved by using FlushMode.AUTO_FLUSH_BACKGROUND, and use checkpoint to ensure at-least-once. Closes #50
* resolve eventual consistency issues * improve speed special on eventual consistency stream * Update Readme
Sometimes we don't want to upsert all columns of a kudu table. So we need to support the function that upsert part of columns of a kudu table.
Fix the problem "Caused by: java.lang.IllegalStateException: Cannot proceed, the client has already been closed" when running in Flink local mode
Kudu connector rework including the addition of a connector to the Table API for it. Co-authored-by: Gyula Fora <[email protected]> Co-authored-by: Balazs Varga <[email protected]>
Update the KuduTableSource to inherit from InputFormatTableSource in order to support both streaming SQL and Batch SQL at the same time. In order to reduce unnecessary data transmission, the filter push down was also added to the KuduTableSource.
Co-authored-by: Gyula Fora <[email protected]>
ferenc-csaky
force-pushed
the
FLINK-34930
branch
from
e15c237
to
121eecb
Compare
|
I am not sure about the current failure with the 1.18.1 + JDK17 combo, I could not reproduce it locally, maybe there is some flakiness here. I'll check a bit deeper soon. |
|
After I took a look at the logs, the error was environmental, it failed to connect to the Dockerized Kudu: @MartijnVisser can that one case be retriggered? Or of not, I hope another run would be green. |
Re-triggered all failing jobs! |
|
Thanks, it passed now! is it okay to merge this change at this point for now? |
ferenc-csaky
changed the title
There was a problem hiding this comment.
@ferenc-csaky I was double checking for legal implications (since this is coming from the Bahir project, that's in the attic).
Looking at https://issues.apache.org/jira/browse/LEGAL-584?focusedCommentId=17402556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17402556 I think what's missing in this PR are the actual LICENSE and NOTICE files. WDYT?
|
@ferenc-csaky I've asked the Legal team for a confirmation on this topic, since the ASF owns both Bahir and Flink code. You can track https://issues.apache.org/jira/browse/LEGAL-675 |
Thank you for taking the time and look into it! For the time being, I migrated the Bahir header to the NOTICE. If it is okay to change it, I'll remove that commit. |
There was a problem hiding this comment.
I've compared this PR to https://github.com/apache/bahir-flink/commits/master/flink-connector-kudu and it looks complete; https://issues.apache.org/jira/browse/LEGAL-675 also confirmed that we didn't do anything with the NOTICE files. LGTM
For this step, I tried to not introduce changes in the ported logic. I separated semantically different changes into multiple commits, so the [FLINK-34930] prefixed commits summarize the actual changes.
The spotless and checkstyle changes are fairly big, but most of the stuff were about adding javadoc and code/import reformat.
TODO